Case Study: Movies Dataset
Please note that you will need to download the dataset from the course website.
You can find the data at https://junyounglim.github.io/. Please unzip the file at a filepath of your choice.
Here are instructions on how to unzip a file in Windows: https://support.microsoft.com/en-us/help/14200/windows-compress-uncompress-zip-files. For Macs, simply double-click on the file.
Use Pandas to Read the Dataset
The dataset contains about 5000 movies.
The following are the features: budget, genres, homepage, id, keywords, original_language, original_title, overview, popularity, production_companies, production_countries, release_date, revenue, runtime, spoken_languages, status, tagline, title, vote_average, vote_count
Using the read_csv function in pandas, we will transfer this information into our code.
# import pandas and load data
import pandas as pd
filepath = 'tmdb_5000_movies.csv'
movies = pd.read_csv(filepath)
# or
movies = pd.read_csv('tmdb_5000_movies.csv')
# Now that we have the dataset we will start to get a feeling for its layout
movies.head(5)
budget | genres | homepage | id | keywords | original_language | original_title | overview | popularity | production_companies | production_countries | release_date | revenue | runtime | spoken_languages | status | tagline | title | vote_average | vote_count | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 237000000 | [{"id": 28, "name": "Action"}, {"id": 12, "nam... | http://www.avatarmovie.com/ | 19995 | [{"id": 1463, "name": "culture clash"}, {"id":... | en | Avatar | In the 22nd century, a paraplegic Marine is di... | 150.437577 | [{"name": "Ingenious Film Partners", "id": 289... | [{"iso_3166_1": "US", "name": "United States o... | 12/10/09 | 2787965087 | 162.0 | [{"iso_639_1": "en", "name": "English"}, {"iso... | Released | Enter the World of Pandora. | Avatar | 7.2 | 11800 |
1 | 300000000 | [{"id": 12, "name": "Adventure"}, {"id": 14, "... | http://disney.go.com/disneypictures/pirates/ | 285 | [{"id": 270, "name": "ocean"}, {"id": 726, "na... | en | Pirates of the Caribbean: At World's End | Captain Barbossa, long believed to be dead, ha... | 139.082615 | [{"name": "Walt Disney Pictures", "id": 2}, {"... | [{"iso_3166_1": "US", "name": "United States o... | 5/19/07 | 961000000 | 169.0 | [{"iso_639_1": "en", "name": "English"}] | Released | At the end of the world, the adventure begins. | Pirates of the Caribbean: At World's End | 6.9 | 4500 |
2 | 245000000 | [{"id": 28, "name": "Action"}, {"id": 12, "nam... | http://www.sonypictures.com/movies/spectre/ | 206647 | [{"id": 470, "name": "spy"}, {"id": 818, "name... | en | Spectre | A cryptic message from Bond’s past sends him o... | 107.376788 | [{"name": "Columbia Pictures", "id": 5}, {"nam... | [{"iso_3166_1": "GB", "name": "United Kingdom"... | 10/26/15 | 880674609 | 148.0 | [{"iso_639_1": "fr", "name": "Fran\u00e7ais"},... | Released | A Plan No One Escapes | Spectre | 6.3 | 4466 |
3 | 250000000 | [{"id": 28, "name": "Action"}, {"id": 80, "nam... | http://www.thedarkknightrises.com/ | 49026 | [{"id": 849, "name": "dc comics"}, {"id": 853,... | en | The Dark Knight Rises | Following the death of District Attorney Harve... | 112.312950 | [{"name": "Legendary Pictures", "id": 923}, {"... | [{"iso_3166_1": "US", "name": "United States o... | 7/16/12 | 1084939099 | 165.0 | [{"iso_639_1": "en", "name": "English"}] | Released | The Legend Ends | The Dark Knight Rises | 7.6 | 9106 |
4 | 260000000 | [{"id": 28, "name": "Action"}, {"id": 12, "nam... | http://movies.disney.com/john-carter | 49529 | [{"id": 818, "name": "based on novel"}, {"id":... | en | John Carter | John Carter is a war-weary, former military ca... | 43.926995 | [{"name": "Walt Disney Pictures", "id": 2}] | [{"iso_3166_1": "US", "name": "United States o... | 3/7/12 | 284139100 | 132.0 | [{"iso_639_1": "en", "name": "English"}] | Released | Lost in our world, found in another. | John Carter | 6.1 | 2124 |
Our dataset is loaded and looks ok, but it looks like there's some cleaning that needs to be done. Notice how multiple columns seem to be objects.
import re
def to_list_4(strng):
return [item for index, item in enumerate(strng.split('"')) if (index + 1) % 6 == 0]
def to_list_2(strng):
return [item for index, item in enumerate(strng.split('"')) if (index + 5) % 8 == 0]
def to_list_2_mod(strng):
return [item for index, item in enumerate(strng.split('"')) if (index + 3) % 6 == 0]
movies.genres = movies.genres.apply(to_list_4)
movies.keywords = movies.keywords.apply(to_list_4)
movies.production_companies = movies.production_companies.apply(to_list_2_mod)
movies.production_countries = movies.production_countries.apply(to_list_2)
movies.spoken_languages = movies.spoken_languages.apply(to_list_2)
movies.head()
budget | genres | homepage | id | keywords | original_language | original_title | overview | popularity | production_companies | production_countries | release_date | revenue | runtime | spoken_languages | status | tagline | title | vote_average | vote_count | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 237000000 | [Action, Adventure, Fantasy, Science Fiction] | http://www.avatarmovie.com/ | 19995 | [culture clash, future, space war, space colon... | en | Avatar | In the 22nd century, a paraplegic Marine is di... | 150.437577 | [Ingenious Film Partners, Twentieth Century Fo... | [US, GB] | 12/10/09 | 2787965087 | 162.0 | [en, es] | Released | Enter the World of Pandora. | Avatar | 7.2 | 11800 |
1 | 300000000 | [Adventure, Fantasy, Action] | http://disney.go.com/disneypictures/pirates/ | 285 | [ocean, drug abuse, exotic island, east india ... | en | Pirates of the Caribbean: At World's End | Captain Barbossa, long believed to be dead, ha... | 139.082615 | [Walt Disney Pictures, Jerry Bruckheimer Films... | [US] | 5/19/07 | 961000000 | 169.0 | [en] | Released | At the end of the world, the adventure begins. | Pirates of the Caribbean: At World's End | 6.9 | 4500 |
2 | 245000000 | [Action, Adventure, Crime] | http://www.sonypictures.com/movies/spectre/ | 206647 | [spy, based on novel, secret agent, sequel, mi... | en | Spectre | A cryptic message from Bond’s past sends him o... | 107.376788 | [Columbia Pictures, Danjaq, B24] | [GB, US] | 10/26/15 | 880674609 | 148.0 | [fr, en, es, it, de] | Released | A Plan No One Escapes | Spectre | 6.3 | 4466 |
3 | 250000000 | [Action, Crime, Drama, Thriller] | http://www.thedarkknightrises.com/ | 49026 | [dc comics, crime fighter, terrorist, secret i... | en | The Dark Knight Rises | Following the death of District Attorney Harve... | 112.312950 | [Legendary Pictures, Warner Bros., DC Entertai... | [US] | 7/16/12 | 1084939099 | 165.0 | [en] | Released | The Legend Ends | The Dark Knight Rises | 7.6 | 9106 |
4 | 260000000 | [Action, Adventure, Science Fiction] | http://movies.disney.com/john-carter | 49529 | [based on novel, mars, medallion, space travel... | en | John Carter | John Carter is a war-weary, former military ca... | 43.926995 | [Walt Disney Pictures] | [US] | 3/7/12 | 284139100 | 132.0 | [en] | Released | Lost in our world, found in another. | John Carter | 6.1 | 2124 |
Pandas also provides some basic quantitative functions to understand our data.
movies['budget'].describe()
count 4.803000e+03 mean 2.904504e+07 std 4.072239e+07 min 0.000000e+00 25% 7.900000e+05 50% 1.500000e+07 75% 4.000000e+07 max 3.800000e+08 Name: budget, dtype: float64
movies.describe()
budget | id | popularity | revenue | runtime | vote_average | vote_count | |
---|---|---|---|---|---|---|---|
count | 4.803000e+03 | 4803.000000 | 4803.000000 | 4.803000e+03 | 4753.000000 | 4803.000000 | 4803.000000 |
mean | 2.904504e+07 | 57165.484281 | 21.492301 | 8.226064e+07 | 106.853777 | 6.092172 | 690.217989 |
std | 4.072239e+07 | 88694.614033 | 31.816650 | 1.628571e+08 | 22.614586 | 1.194612 | 1234.585891 |
min | 0.000000e+00 | 5.000000 | 0.000000 | 0.000000e+00 | 0.000000 | 0.000000 | 0.000000 |
25% | 7.900000e+05 | 9014.500000 | 4.668070 | 0.000000e+00 | 94.000000 | 5.600000 | 54.000000 |
50% | 1.500000e+07 | 14629.000000 | 12.921594 | 1.917000e+07 | 104.000000 | 6.200000 | 235.000000 |
75% | 4.000000e+07 | 58610.500000 | 28.313505 | 9.291719e+07 | 117.000000 | 6.800000 | 737.000000 |
max | 3.800000e+08 | 459488.000000 | 875.581305 | 2.787965e+09 | 338.000000 | 10.000000 | 13752.000000 |
movies['vote_average'].mean()
6.092171559442011
movies.mean()
budget 2.904504e+07 id 5.716548e+04 popularity 2.149230e+01 revenue 8.226064e+07 runtime 1.068538e+02 vote_average 6.092172e+00 vote_count 6.902180e+02 dtype: float64
movies.corr()
budget | id | popularity | revenue | runtime | vote_average | vote_count | |
---|---|---|---|---|---|---|---|
budget | 1.000000 | -0.089377 | 0.505414 | 0.730823 | 0.267915 | 0.093146 | 0.593180 |
id | -0.089377 | 1.000000 | 0.031202 | -0.050425 | -0.152379 | -0.270595 | -0.004128 |
popularity | 0.505414 | 0.031202 | 1.000000 | 0.644724 | 0.224205 | 0.273952 | 0.778130 |
revenue | 0.730823 | -0.050425 | 0.644724 | 1.000000 | 0.249986 | 0.197150 | 0.781487 |
runtime | 0.267915 | -0.152379 | 0.224205 | 0.249986 | 1.000000 | 0.377139 | 0.272182 |
vote_average | 0.093146 | -0.270595 | 0.273952 | 0.197150 | 0.377139 | 1.000000 | 0.312997 |
vote_count | 0.593180 | -0.004128 | 0.778130 | 0.781487 | 0.272182 | 0.312997 | 1.000000 |
We can also filter information conditionally.
filter_1 = movies['vote_average'] > 8.0
print(filter_1.any())
movies[movies['vote_average'] > 8.0]
True
budget | genres | homepage | id | keywords | original_language | original_title | overview | popularity | production_companies | production_countries | release_date | revenue | runtime | spoken_languages | status | tagline | title | vote_average | vote_count | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
65 | 185000000 | [Drama, Action, Crime, Thriller] | http://thedarkknight.warnerbros.com/dvdsite/ | 155 | [dc comics, crime fighter, secret identity, sc... | en | The Dark Knight | Batman raises the stakes in his war on crime. ... | 187.322927 | [DC Comics, Legendary Pictures, Warner Bros., ... | [GB, US] | 7/16/08 | 1004558444 | 152.0 | [en, zh] | Released | Why So Serious? | The Dark Knight | 8.2 | 12002 |
95 | 165000000 | [Adventure, Drama, Science Fiction] | http://www.interstellarmovie.net/ | 157336 | [saving the world, artificial intelligence, fa... | en | Interstellar | Interstellar chronicles the adventures of a gr... | 724.247784 | [Paramount Pictures, Legendary Pictures, Warne... | [CA, US, GB] | 11/5/14 | 675120017 | 169.0 | [en] | Released | Mankind was born on Earth. It was never meant ... | Interstellar | 8.1 | 10867 |
96 | 160000000 | [Action, Thriller, Science Fiction, Mystery, A... | http://inceptionmovie.warnerbros.com/ | 27205 | [loss of lover, dream, kidnapping, sleep, subc... | en | Inception | Cobb, a skilled thief who commits corporate es... | 167.583710 | [Legendary Pictures, Warner Bros., Syncopy] | [GB, US] | 7/14/10 | 825532764 | 148.0 | [en, ja, fr] | Released | Your mind is the scene of the crime. | Inception | 8.1 | 13752 |
329 | 94000000 | [Adventure, Fantasy, Action] | http://www.lordoftherings.net | 122 | [elves, orcs, middle-earth (tolkien), based on... | en | The Lord of the Rings: The Return of the King | Aragorn is revealed as the heir to the ancient... | 123.630332 | [WingNut Films, New Line Cinema] | [NZ, US] | 12/1/03 | 1118888979 | 201.0 | [en] | Released | The eye of the enemy is moving. | The Lord of the Rings: The Return of the King | 8.1 | 8064 |
662 | 63000000 | [Drama] | http://www.foxmovies.com/movies/fight-club | 550 | [support group, dual identity, nihilism, rage ... | en | Fight Club | A ticking-time-bomb insomniac and a slippery s... | 146.757391 | [Regency Enterprises, Fox 2000 Pictures, Tauru... | [DE, US] | 10/15/99 | 100853753 | NaN | [en] | Released | Mischief. Mayhem. Soap. | Fight Club | 8.3 | 9413 |
690 | 60000000 | [Fantasy, Drama, Crime] | http://thegreenmile.warnerbros.com/ | 497 | [southern usa, black people, mentally disabled... | en | The Green Mile | A supernatural tale set on death row in a Sout... | 103.698022 | [Castle Rock Entertainment, Darkwoods Producti... | [US] | 12/10/99 | 284600000 | 189.0 | [fr, en] | Released | Miracles do happen. | The Green Mile | 8.2 | 4048 |
809 | 55000000 | [Comedy, Drama, Romance] | NaN | 13 | [vietnam veteran, hippie, mentally disabled, r... | en | Forrest Gump | A man with a low IQ has accomplished great thi... | 138.133331 | [Paramount Pictures] | [US] | 7/6/94 | 677945399 | 142.0 | [en] | Released | The world will never be the same, once you've ... | Forrest Gump | 8.2 | 7927 |
1553 | 33000000 | [Crime, Mystery, Thriller] | http://www.sevenmovie.com/ | 807 | [self-fulfilling prophecy, detective, s.w.a.t.... | en | Se7en | Two homicide detectives are on a desperate hun... | 79.579532 | [New Line Cinema, Juno Pix, Cecchi Gori Pictures] | [US] | 9/22/95 | 327311859 | 127.0 | [en] | Released | Seven deadly sins. Seven ways to die. | Se7en | 8.1 | 5765 |
1663 | 30000000 | [Drama, Crime] | NaN | 311 | [life and death, corruption, street gang, rape... | en | Once Upon a Time in America | A former Prohibition-era Jewish gangster retur... | 49.336397 | [Warner Bros., The Ladd Company] | [US, IT] | 2/16/84 | 0 | 229.0 | [en, fr, it] | Released | Crime, passion and lust for power - Sergio Leo... | Once Upon a Time in America | 8.2 | 1069 |
1818 | 22000000 | [Drama, History, War] | http://www.schindlerslist.com/ | 424 | [factory, concentration camp, hero, holocaust,... | en | Schindler's List | The true story of how businessman Oskar Schind... | 104.469351 | [Universal Pictures, Amblin Entertainment] | [US] | 11/29/93 | 321365567 | 195.0 | [de, pl, he, en] | Released | Whoever saves one life, saves the world entire. | Schindler's List | 8.3 | 4329 |
1847 | 25000000 | [Drama, Crime] | http://www.warnerbros.com/goodfellas | 769 | [prison, based on novel, florida, 1970s, mass ... | en | GoodFellas | The true story of Henry Hill, a half-Irish, ha... | 63.654244 | [Winkler Films] | [US] | 9/12/90 | 46836394 | 145.0 | [it, en] | Released | Three Decades of Life in the Mafia. | GoodFellas | 8.2 | 3128 |
1881 | 25000000 | [Drama, Crime] | NaN | 278 | [prison, corruption, police brutality, prison ... | en | The Shawshank Redemption | Framed in the 1940s for the double murder of h... | 136.747729 | [Castle Rock Entertainment] | [US] | 9/23/94 | 28341469 | 142.0 | [en] | Released | Fear can hold you prisoner. Hope can set you f... | The Shawshank Redemption | 8.5 | 8205 |
1987 | 24000000 | [Fantasy, Animation, Adventure] | NaN | 4935 | [flying, witch, rain, castle, scarecrow, body ... | ja | ハウルの動く城 | When Sophie, a shy young woman, is cursed with... | 49.549984 | [Studio Ghibli, Buena Vista Home Entertainment... | [JP] | 11/19/04 | 234710455 | 119.0 | [ja] | Released | The two lived there | Howl's Moving Castle | 8.2 | 1991 |
1990 | 18000000 | [Adventure, Action, Science Fiction] | http://www.starwars.com/films/star-wars-episod... | 1891 | [rebel, android, asteroid, space battle, snow ... | en | The Empire Strikes Back | The epic saga continues as Luke Skywalker, in ... | 78.517830 | [Lucasfilm, Twentieth Century Fox Film Corpora... | [US] | 5/17/80 | 538400000 | 124.0 | [en] | Released | The Adventure Continues... | The Empire Strikes Back | 8.2 | 5879 |
2091 | 19000000 | [Crime, Drama, Thriller] | NaN | 274 | [based on novel, psychopath, horror, suspense,... | en | The Silence of the Lambs | FBI trainee, Clarice Starling ventures into a ... | 18.174804 | [Orion Pictures, Strong Heart/Demme Production] | [US] | 2/1/91 | 272742922 | 119.0 | [en] | Released | To enter the mind of a killer she must challen... | The Silence of the Lambs | 8.1 | 4443 |
2170 | 806948 | [Drama, Horror, Thriller] | NaN | 539 | [hotel, clerk, arizona, shower, rain, motel, m... | en | Psycho | When larcenous real estate clerk Marion Crane ... | 93.067866 | [Paramount Pictures, Universal Pictures, Shaml... | [US] | 6/16/60 | 32000000 | 109.0 | [en] | Released | The master of suspense moves his cameras into ... | Psycho | 8.2 | 2320 |
2247 | 26500000 | [Adventure, Fantasy, Animation] | NaN | 128 | [fight, wolf, village and town, iron, pan, wil... | ja | もののけ姫 | Ashitaka, a prince of the disappearing Ainu tr... | 60.732738 | [Miramax Films, Studio Ghibli, Nibariki, Nippo... | [JP] | 7/12/97 | 159375308 | 134.0 | [ja] | Released | The Fate Of The World Rests On The Courage Of ... | Princess Mononoke | 8.2 | 1983 |
2284 | 19000000 | [Horror, Thriller] | NaN | 694 | [hotel, isolation, hotelier, colorado, maze, b... | en | The Shining | Jack Torrance accepts a caretaker job at the O... | 78.699993 | [Hawk Films, Warner Bros., Peregrine] | [US, GB] | 5/22/80 | 44017374 | 144.0 | [en] | Released | A masterpiece of modern horror. | The Shining | 8.1 | 3757 |
2294 | 15000000 | [Fantasy, Adventure, Animation, Family] | http://movies.disney.com/spirited-away | 129 | [witch, parents kids relationship, magic, twil... | ja | 千と千尋の神隠し | A ten year old girl who wanders away from her ... | 118.968562 | [Studio Ghibli] | [JP] | 7/20/01 | 274925095 | 125.0 | [ja] | Released | The tunnel led Chihiro to a mysterious town... | Spirited Away | 8.3 | 3840 |
2386 | 0 | [Western, Action, Drama, History] | NaN | 69848 | [war, army, battlefield, chivalry] | en | One Man's Hero | One Man's Hero tells the little-known story of... | 0.910529 | [Filmax, Silver Lion Films, Televisa S.A. de C... | [US, MX, ES] | 8/2/99 | 0 | 121.0 | [en] | Released | One man's hero is another man's traitor. | One Man's Hero | 9.3 | 2 |
2453 | 16400000 | [Drama] | NaN | 207 | [individual, philosophy, poetry, shakespeare, ... | en | Dead Poets Society | At an elite, old-fashioned boarding school in ... | 55.458584 | [Touchstone Pictures, Silver Screen Partners IV] | [US] | 6/2/89 | 235860116 | 129.0 | [en] | Released | He was their inspiration. He made their lives ... | Dead Poets Society | 8.1 | 2705 |
2731 | 13000000 | [Drama, Crime] | NaN | 240 | [italo-american, cuba, vororte, melancholy, pr... | en | The Godfather: Part II | In the continuing saga of the Corleone crime f... | 105.792936 | [Paramount Pictures, The Coppola Company] | [US] | 12/20/74 | 47542841 | 200.0 | [en, it, la, es] | Released | I don't feel I have to wipe everybody out, Tom... | The Godfather: Part II | 8.3 | 3338 |
2760 | 6000000 | [Drama, Thriller] | http://www.roomthemovie.com | 264644 | [based on novel, carpet, isolation, kidnapping... | en | Room | Jack is a young boy of 5 years old who has liv... | 66.113340 | [T\u00e9l\u00e9film Canada, Film 4, Element Pi... | [CA, IE] | 10/16/15 | 35401758 | 117.0 | [en] | Released | Love knows no boundaries | Room | 8.1 | 2757 |
2796 | 0 | [Adventure, Drama, Romance] | NaN | 43867 | [kidnapping, coronation, villain, kingdom, hei... | en | The Prisoner of Zenda | An Englishman on a Ruritarian holiday must imp... | 4.021389 | [United Artists, Selznick International Pictures] | [US] | 9/3/37 | 0 | NaN | [es, en] | Released | The most thrilling swordfight ever filmed... | The Prisoner of Zenda | 8.4 | 11 |
2912 | 11000000 | [Adventure, Action, Science Fiction] | http://www.starwars.com/films/star-wars-episod... | 11 | [android, galaxy, hermit, death star, lightsab... | en | Star Wars | Princess Leia is captured and held hostage by ... | 126.393695 | [Lucasfilm, Twentieth Century Fox Film Corpora... | [US] | 5/25/77 | 775398007 | 121.0 | [en] | Released | A long time ago in a galaxy far, far away... | Star Wars | 8.1 | 6624 |
2947 | 10000000 | [Drama, History] | NaN | 30973 | [miracle, jesus christ, last supper, bible, ch... | en | The Visual Bible: The Gospel of John | A word for word depiction of the life of Jesus... | 3.208172 | [Gospel of John Ltd., Toronto Film Studios, Vi... | [CA, GB] | 9/11/03 | 4069090 | 125.0 | [en] | Released | For God loved the world So much... | The Visual Bible: The Gospel of John | 8.2 | 12 |
2970 | 10500000 | [Drama, Comedy] | NaN | 88641 | [] | en | There Goes My Baby | A group of high school seniors meets in the su... | 0.037073 | [Nelson Entertainment] | [US] | 9/2/94 | 123509 | 99.0 | [en] | Released | NaN | There Goes My Baby | 8.5 | 2 |
3041 | 10000000 | [Comedy, Drama, Romance] | NaN | 27322 | [sex, ex-boyfriend, independent film, african ... | en | Love Jones | Darius Lovehall is a young black poet in Chica... | 1.000178 | [New Line Cinema, Addis Wechsler Pictures] | [US] | 3/14/97 | 0 | 104.0 | [en] | Released | Get Together. Fall Apart. Start Over. | Love Jones | 8.1 | 12 |
3057 | 20000000 | [Drama] | http://www.historyx.com/ | 73 | [usa, neo-nazi, prison, skinhead, brother brot... | en | American History X | Derek Vineyard is paroled after serving 3 year... | 73.567232 | [New Line Cinema, The Turman-Morrissey Company... | [US] | 10/30/98 | 23875127 | 119.0 | [en] | Released | Some Legacies Must End. | American History X | 8.2 | 3016 |
3232 | 8000000 | [Thriller, Crime] | NaN | 680 | [transporter, brothel, drug dealer, boxer, mas... | en | Pulp Fiction | A burger-loving hit man, his philosophical par... | 121.463076 | [Miramax Films, A Band Apart, Jersey Films] | [US] | 10/8/94 | 213928762 | 154.0 | [en, es, fr] | Released | Just because you are a character doesn't mean ... | Pulp Fiction | 8.3 | 8428 |
3337 | 6000000 | [Drama, Crime] | http://www.thegodfather.com/ | 238 | [italy, love at first sight, loss of father, p... | en | The Godfather | Spanning the years 1945 to 1955, a chronicle o... | 143.659698 | [Paramount Pictures, Alfran Productions] | [US] | 3/14/72 | 245066411 | 175.0 | [en, it, la] | Released | An offer you can't refuse. | The Godfather | 8.4 | 5893 |
3454 | 6000000 | [Drama, Crime, Thriller] | http://www.mgm.com/#/our-titles/2083/The-Usual... | 629 | [law, relatives, theft, criminal, criminal mas... | en | The Usual Suspects | Held in an L.A. interrogation room, Verbal Kin... | 64.025031 | [Blue Parrot Productions, Bad Hat Harry Produc... | [US] | 7/19/95 | 23341568 | NaN | [es, en, fr, hu] | Released | Five Criminals. One Line Up. No Coincidence. | The Usual Suspects | 8.1 | 3254 |
3519 | 0 | [Comedy] | NaN | 89861 | [italy, victorian england, young woman] | en | Stiff Upper Lips | Stiff Upper Lips is a broad parody of British ... | 0.356495 | [] | [GB, IN] | 6/12/98 | 0 | 99.0 | [en] | Released | NaN | Stiff Upper Lips | 10.0 | 1 |
3573 | 9000000 | [Mystery, Thriller] | http://www.otnemem.com/ | 77 | [individual, insulin, tattoo, waitress, amnesi... | en | Memento | Suffering short-term memory loss after a head ... | 60.715151 | [Summit Entertainment, Newmarket Capital Group... | [US] | 10/11/00 | 39723096 | 113.0 | [en] | Released | Some memories are best forgotten. | Memento | 8.1 | 4028 |
3622 | 5000000 | [Western] | NaN | 335 | [showdown, bounty, bounty hunter, loss of brot... | it | C'era una volta il West | This classic western masterpiece is an epic fi... | 49.333137 | [Paramount Pictures, Rafran Cinematografica, F... | [IT, ES, US] | 12/21/68 | 5321508 | 175.0 | [en] | Released | There were three men in her life. One to take ... | Once Upon a Time in the West | 8.1 | 1128 |
3719 | 3000000 | [Drama] | NaN | 510 | [individual, rebel, self-destruction, wheelcha... | en | One Flew Over the Cuckoo's Nest | While serving time for insanity at a state men... | 127.525581 | [United Artists, Fantasy Films, Warner Bros.] | [US] | 11/18/75 | 108981275 | 133.0 | [en] | Released | If he's crazy, what does that make you? | One Flew Over the Cuckoo's Nest | 8.2 | 2919 |
3723 | 0 | [Drama, Family] | http://www.anneofgreengables.com/ | 17663 | [based on novel, brother sister relationship, ... | en | Anne of Green Gables | At the turn of the century on Prince Edward Is... | 8.772574 | [Sullivan Entertainment] | [US] | 12/1/85 | 0 | 199.0 | [en] | Released | NaN | Anne of Green Gables | 8.2 | 68 |
3788 | 4000000 | [Drama] | NaN | 242575 | [germany, mexican] | en | Guten Tag, Ramón | After five failed attempts to go to the United... | 1.771584 | [Eficine 226, Beanca Films, Fondo de Inversi\u... | [MX, DE] | 10/18/13 | 0 | 119.0 | [de, es] | Released | NaN | Guten Tag, Ramón | 8.1 | 18 |
3865 | 3300000 | [Drama] | http://sonyclassics.com/whiplash/ | 244786 | [jazz, obsession, conservatory, music teacher,... | en | Whiplash | Under the direction of a ruthless instructor, ... | 192.528841 | [Bold Films, Blumhouse Productions, Right of W... | [US] | 10/10/14 | 13092000 | 105.0 | [en] | Released | The road to greatness can take you to the edge. | Whiplash | 8.3 | 4254 |
3866 | 3300000 | [Drama, Crime] | http://cidadededeus.globo.com/ | 598 | [male nudity, street gang, brazilian, photogra... | pt | Cidade de Deus | Cidade de Deus is a shantytown that started du... | 44.356711 | [O2 Filmes, VideoFilmes, Wild Bunch, Globo fil... | [BR, FR] | 2/5/02 | 30641770 | 130.0 | [pt] | Released | If you run you're dead... if you stay, you're ... | City of God | 8.1 | 1814 |
3906 | 3000000 | [Comedy, Drama, Romance] | NaN | 284 | [new york, new year's eve, lovesickness, age d... | en | The Apartment | Bud Baxter is a minor clerk in a huge New York... | 22.889294 | [United Artists, The Mirisch Company] | [US] | 6/15/60 | 25000000 | 125.0 | [en] | Released | Movie-wise, there has never been anything like... | The Apartment | 8.1 | 483 |
3992 | 0 | [] | NaN | 346081 | [] | en | Sardaarji | A ghost hunter uses bottles to capture trouble... | 0.296981 | [] | [IN] | 6/26/15 | 0 | 0.0 | [] | Released | NaN | Sardaarji | 9.5 | 2 |
4045 | 0 | [Comedy, Drama, Family] | NaN | 78373 | [small town, texas] | en | Dancer, Texas Pop. 81 | Four guys, best friends, have grown up togethe... | 0.376662 | [HSX Films, Chase Productions] | [US] | 5/1/98 | 565592 | 97.0 | [en] | Released | in the middle of nowhere they had everything | Dancer, Texas Pop. 81 | 10.0 | 1 |
4238 | 1 | [Drama, Comedy] | NaN | 3082 | [factory, ambulance, invention, tramp, great d... | en | Modern Times | The Tramp struggles to live in modern industri... | 28.276480 | [United Artists, Charles Chaplin Productions] | [US] | 2/5/36 | 8500000 | 87.0 | [en] | Released | He stands alone as the greatest entertainer of... | Modern Times | 8.1 | 856 |
4247 | 1 | [Romance, Comedy, Drama] | NaN | 361505 | [] | en | Me You and Five Bucks | A womanizing yet lovable loser, Charlie, a wai... | 0.094105 | [] | [] | 7/7/15 | 0 | 90.0 | [] | Released | A story about second, second chances | Me You and Five Bucks | 10.0 | 2 |
4302 | 1200000 | [Western] | http://www.mgm.com/#/our-titles/766/The-Good,-... | 429 | [bounty hunter, refugee, gold, anti hero, gall... | it | Il buono, il brutto, il cattivo | While the Civil War rages between the Union an... | 88.377076 | [United Artists, Constantin Film Produktion, P... | [US, IT, ES, DE] | 12/23/66 | 6000000 | 161.0 | [it] | Released | For three men the Civil War wasn't hell. It wa... | The Good, the Bad and the Ugly | 8.1 | 2311 |
4535 | 2000000 | [Action, Drama] | NaN | 346 | [japan, samurai, peasant, looting, rice, fenci... | ja | 七人の侍 | A samurai answers a village's request for prot... | 39.756748 | [Toho Company] | [JP] | 4/26/54 | 271841 | 207.0 | [ja] | Released | The Mighty Warriors Who Became the Seven Natio... | Seven Samurai | 8.2 | 878 |
4602 | 350000 | [Drama] | NaN | 389 | [judge, jurors, sultriness, death penalty, fat... | en | 12 Angry Men | The defense and the prosecution have rested an... | 59.259204 | [United Artists, Orion-Nova Productions] | [US] | 3/25/57 | 1000000 | 96.0 | [en] | Released | Life is in their hands. Death is on their minds. | 12 Angry Men | 8.2 | 2078 |
4662 | 0 | [Comedy] | NaN | 40963 | [independent film] | en | Little Big Top | An aging out of work clown returns to his smal... | 0.092100 | [Fly High Films] | [US] | 1/1/06 | 0 | 0.0 | [en] | Rumored | NaN | Little Big Top | 10.0 | 1 |
4755 | 50000 | [Documentary] | NaN | 322745 | [] | en | Counting | An associative collection of visual impression... | 0.293587 | [] | [US] | 2/9/15 | 0 | 111.0 | [en] | Released | NaN | Counting | 8.3 | 3 |
filter_2 = movies['vote_average'] > 8.0
filter_2.all()
False
movies.shape
(4803, 20)
#Check if there are Null values in each row
movies.isnull().any()
budget False genres False homepage True id False keywords False original_language False original_title False overview True popularity False production_companies False production_countries False release_date True revenue False runtime True spoken_languages False status False tagline True title False vote_average False vote_count False dtype: bool
Let's start with fixing the categorical NaN values.
movies_filled = movies
movies_filled['homepage'] = movies_filled['homepage'].fillna(value='None')
movies_filled['overview'] = movies_filled['overview'].fillna(value='')
movies_filled['tagline'] = movies_filled['tagline'].fillna(value='')
Now let's do the numerical NaN values.
print(movies_filled['runtime'].isnull().sum())
print(movies_filled['release_date'].isnull().sum())
50 1
movies_filled['runtime'] = movies_filled['runtime'].fillna(value=movies_filled['runtime'].median())
movies_filled = movies_filled.dropna()
movies_filled.isnull().any()
budget False genres False homepage False id False keywords False original_language False original_title False overview False popularity False production_companies False production_countries False release_date False revenue False runtime False spoken_languages False status False tagline False title False vote_average False vote_count False dtype: bool
Thats nice! No NULL rows!